Commentary on "The Optimality of Jeffreys Prior for Online Density Estimation and the Asymptotic Normality of Maximum Likelihood Estimators"

In the field of prediction with expert advice, a standard goal is to sequentially predict data as well as the best expert in some reference set of ‘expert predictors’. Universal data compression, a subfield of information theory, can be thought of as a special case. Here, the set of expert predictors is a statistical model, i.e. a family of probability distributions, and the predictions are scored using the logarithmic loss function, which, via the Kraft inequality, gives the procedure an interpretation in terms of data compression. A prediction strategy is a function that, for each n, given data xn ≡ x1, . . . , xn, outputs a “predictive” probability distribution p(· | xn) for Xi+1. For a given modelM, the Shtarkov or Normalized Maximum Likelihood (NML) strategy relative to M, is the prediction strategy that achieves the minimax optimal individual-sequence regret relative to M. NML has a number of drawbacks, detailed below, and is therefore often approximated by more convenient strategies such as Sequential Normalized Maximum Likelihood (SNML) or the Bayesian strategy. The latter predicts using the Bayesian predictive distribution for the model M, defined relative to some prior π, which is often taken to be Jeffreys’ prior — in that case we abbreviate it to J.B. The text below has been written so as to be (hopefully) understandable for readers who do not know too many details of these concepts; for such details, see e.g. Grünwald (2007) and/or Kotlowski and Grünwald (2011) (KG from now on).